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(54) Expert system for presenting multimedia information in a tree structure. 

(57) The system uses multimedia information tools (i.e. graphics, documentation, drawings, photographs, 
full motion video with audio, PLC ladder, etc.) and techniques to assist maintenance personnel. Diagnostic 
development and multimedia information assignment are accomplished through graphical visual 
programming, which requires no conventional software programming. The system uses diagnostic trees 
having multimedia graphic icons which define the diagnostic components. In developing a visual diagnostic 
tree, the multimedia information can be assigned or mapped to each node of the tree by selecting an 
appropriate icon to represent the multimedia from an icon library. Then, an appropriate media information file 
or record and comments for that icon are added. The system uses conventional hardware and can be used on 
the plant floor. 
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METHOD AND SYSTEM FOR PROCESSING AND PRESENTING 
ON-LINE, MULTIMEDIA INFORMATION IN A TREE STRUCTURE 

5 This invention relates to methods and systems for 

processing and presenting multimedia information and, in 
particular, to methods and systems for processing and 
presenting on-line, multimedia information such as 
diagnostic information for a machine tool in a tree 

10 structure. 

Reference is made to an application entitled "Method 
And System For Diagnosing Machines" filed on the same day as 
this application, having the same assignee, and which is 
hereby expressly incorporated in its entirety by reference. 

15 There exists a need for timely delivery of diagnostic 

information for machine tools such as stamping presses. As 
illustrated in Figure 1, the need is the greatest when the 
press is first delivered. As time passes, the plant 
personnel become more experienced and less dependent on a 

20 diagnostic system. The key is continually to have the 

proper tools present to minimize press down time, regardless 
of the need. 

Based on these observations, the machine tool 
manufacturer must develop the diagnostic aid as the press is 

25 being designed if this goal is going to be achieved. 

Current expert systems and/or artificial intelligence 
technology tools can not meet the above direction, goals and 
design needs. The deficiency is in the ability to allow end 
users to build and maintain diagnostics, especially as the 

30 plant machinery goes through modification. Based on the 
current state of software/hardware technology (i.e. object 
oriented) , there exists a need for a new method and system 
which uses this technology in a novel way for capturing, 
preserving and delivering machine tool diagnostic and 

35 support information. 

The U.S. patent to Isle et al. (4,931,950) discloses a 
multimedia interface method for a knowledge-based diagnostic 
system. The knowledge-based system stores multimedia 
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commands to control multimedia outputs for presenting 

information to users. 

The U.S. patent to Tsumura (4,954,969) discloses a 

system for processing multimedia information in a easily 
5 usable form. 

The U.S. patent to Singh (4,954,964) discloses an 

apparatus and method for expert analysis of metal failure 

with automated visual aid. The invention combines an expert 

system with a video or photographic display system. 
10 The U.S. patent to Rutherfoord et al. (4,893,256) 

discloses an interactive multimedia presentation system and 

a method for developing the presentation. 

The U.S. patent to Bodick et al. (4,945,476) discloses 

a knowledge-base system with stored pictorial images for use 
15 in medical diagnostic environments. 

The prior art discloses the use of icons in various 

systems. For example, the U.S. patent to Nose et al. 

(5,025,395) discloses a data processing system which employs 

icons to represent various functions of the system. 
20 The U.S. patent to Wexelblat et al. (5,021,976) 

discloses a method and system for generating dynamic, 

interactive visual representations of information 

structures. The patent teaches the use of iconic 

representations in an information system. 
25 The U.S. patents to Dunn (4,656,603 and 4,813,013) 

disclose interactive rule based systems employing icons. 

The system generates and stores a specific set of rules 

pertaining to the use of each icon. 

The paper entitled "The Relationship Between Multimedia 
3 0 And Expert Systems" by John Coyne, discloses a philosophy to 

using an expert system and multimedia but fails to provide 

many details. 

The U.S. patent to Franke et al. (4,710,763) discloses 
a method of constructing and developing a tree structure. 
35 An operator is capable of performing editing and evaluating 
functions on the tree. 

The U.S. patent to Watson et al. (4,902,469) discloses 
apparatus for producing a discrete state display including 
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status trees and status tree node operations. 

An object of the present invention is to provide a 
method and system for processing and presenting on-line, 
multimedia information in a tree structure. 
5 Another object of the present invention is to provide a 

relatively simple and flexible method and system for 
processing and presenting on-line, multimedia information 
such as diagnostic information for a machine tool wherein 
initial diagnostic knowledge and documentation (such as 
10 drawings) are electronically captured as designers are 
designing the machine tool, which eliminates any possible 
loss or re-engineering of this information at a later time. 

According to the present invention there is provided 
a method for processing and presenting multimedia 
15 information in a tree structure including branches having a 
plurality of nodes and node-connecting links in a computer 
system having tree display means for displaying nodes of the 
tree structure, the method comprising the steps of: 

providing command display means for displaying 
20 multimedia commands representative of various types of 
multimedia information; 

providing multimedia output means for providing 
multimedia information to a user of the: system; 

providing a. knowledge base for storing a 
25 multiplicity of information records mapped into the tree 
structure, the information records including: 

means for denoting the text strings that can be 
displayed on the tree display means; 

means for denoting the multimedia commands that 
30 can be displayed on the command display means; and 

means for denoting the multimedia information that 
can be presented by said multimedia output means; 
receiving an initial command; 

displaying at least one selected node of the tree 
35 structure including at least one text string on the tree 
display means in response to the initial command; 

displaying a plurality of multimedia commands 
corresponding to the at least one selected node on the 
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command display means; 

receiving a user command related to one of the 
displayed multimedia commands; and 

delivering a selected amount of the multimedia 
5 information to the multimedia output means to be presented 
thereby to the user of the system in response to the user 
command . 

A system is also provided for carrying out each of the 
above method steps. 
10 Preferably, the multimedia commands are displayed on 

the command display in the form of graphic icons . 

The invention will now be described further by way of 
example, with reference to the accompanying drawings, in 
which: 

15 Figure 1 is a graph illustrating the interrelationship 

of the need for diagnostic information to press development 
time and press operational time; 

Figure 2 is a schematic diagram illustrating the 
architecture of the system of the present invention 
20 including a diagnostic system coupled to a database of 
machine-specific knowledge and support informations- 
Figure 3 is a schematic diagram of a diagnostic tree 
and multimedia mapping of the present inventions- 
Figure 4 is a schematic view of the hardware of the 
2 5 system; 

Figure 5 is a schematic view of the software of the 
systems- 
Figure 6 is a schematic block diagram of a diagnostic 
tree and presentation server structure of the present 
30 inventions- 
Figure 7 is a schematic view of an example question and 
answer (Q and A) screen in text formats- 
Figure 8 is a schematic view of an example question and 
answer screen in graphics-with-text format ; 
35 Figure 9 is a schematic view of a presentation server 

screens- 
Figure 10 is a schematic view of a diagnostic branch 
(edit) screen; 
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Figure 11 is a graphical view of a presentation server 
edit screen or multimedia definition screen; 

Figure 12 is a graphical view of a screen illustrating 
diagnostic tree development with multimedia icon edit; 
5 Figure 13 is a graphical view of a media selection 

server; 

Figure 14 is a graphical view of an initial application 
screen ; 

Figure 15 is a graphical view of a standard Q and A 
10 screen similar to Figure 8; 

Figure 16 is a graphical view of a multimedia view 
screen similar to Figure 9 with video selected; 

Figure 17 is a graphical view of a documentation viev; 
screen; 

15 Figure 18 is a graphical view of a screen for a 

drawing; 

Figure 19 is a graphical view of a screen for still 
video ; 

Figure 20 is a graphical view of a screen for full 

20 motion values; and 

Figure 21 is a graphical view of a solution screen. 
Referring now to the drawing figures, there is 
schematically illustrated in Figure 2 a system including a 
generic core of software and hardware, which is referred to 

25 below as a diagnostic system 20 for use with a machine tool 
such as a press. To give life to the diagnostic system 20 
for a unique machine tool diagnostic solution requires 
specific machine tool diagnostic knowledge and support 
information (referred to below as a database 22) . 

30 The architecture of the system illustrated in Figure 2, 

shows three primary components of the system: the 
diagnostic system 20, a knowledge acquisition tool 24, and 
the database 22. 

The diagnostic system 20 "is dependent on the data to 

35 enable it as an application. The data in the database 22 
includes machine tool specific diagnostic knowledge and 
support information (i.e. drawings and video) which is 
stored in the form of information records. The machine tool 
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support information provides alternative methods of 
communication and documentation necessary to properly inform 
the end user while doing his or her job. Another important 
aspect of this architecture is that the machine specific 

5 support information can be stored in its original format 
(for example, WordPerfect). There is no need to translate 
or re-engineer the information, thus reducing development 
time, expense and effort. 

The knowledge acquisition tool 24 of the present 

10 invention is relatively simple and intuitive to use; 

provides graphical representation of diagnostics knowledge; 
and provides mapping of machine tool related support 
information . 

The tool 24 represents the knowledge in graphical trees 
15 (or diagnostic trees). Within a tree there are several 

questions and answers, and solution nodes as illustrated in 
Figure 3. Nodes can be grouped together into branches, 
making the diagnostic tree more modular and reusable. There 
is no software programming involved in building the trees or 
2 0 mapping the support knowledge - it is performed through 
graphical visual programming. Training to use of the 
diagnostic tool can be accomplished relatively quickly. 

The procedure is similar to using a computer drawing 
package, by graphically selecting and placing objects 
25 (denoted by boxes) , asserting text (questions & answers) , 
linking boxes and assigning support information to 
respective nodes as illustrated in Figure 3. Training to 
build and maintain diagnostic trees, and assign or edit 
machine specific support information can also be 
30 accomplished relatively quickly. 

The method and system of the present invention 
preferably utilizes an IBM PC platform that operates under 
DOS 5.0 and Windows 3.0. This platform provides a rich 
design environment and option flexibility. 
35 target system preferably includes, at a minimum, an 

IBM£3^5/486 or compatible computer with a 200Mb hard drive, 
8 Mb of RAM, VGA display, Video card, a computer controlled 
VCR, optional speakers, and a direct connection to a press 
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PLC network as illustrated in Figure 4. 

There is illustrated in Figure 5, a software portion of 
the system. The software includes commercially available 
software and one custom piece of softv/are called a 

5 presentation server 26. The Windows 3.0 environment allows 
multiple software package to run concurrently and 
communicate between themselves. Communication between the 
software packages leverages the software packages 
functionality into a integrated solution and minimizes 

10 software development. 

The diagnostic system 20, the knowledge acquisition 
tool 24 and the database 22 are preferably incorporated in a 
software package called "Advisor" provided by Emerald 
Intelligence of Ann Arbor, Michigan. Advisor provides the 

15 environment to build and execute diagnostics. To assign 

(i.e. map) and present (i.e. display) multimedia information 
for each of the diagnostic tree nodes as illustrated in 
Figure 6, Advisor communicates and interacts with the 
presentation server 26. The presentation server 26 then 

20 communicateswith the media specific software packages (i.e. 
Microsof tjj^T^ for documentation media) to select and view. 

With this architecture, it is possible for each 
diagnostic node to have multimedia information assigned or 
mapped thereto thus improving the communication between the 

2 5 operator and the computer. The standard forms of multimedia 

utilized are: drawings, documentations, graphics, 
photographs, full motion video and audio, animation, sound, 
etc. For example, this approach can provide a better and 
easier means of on-line interactive repair and training 
30 procedures (i.e. by using full motion video) to the operator 
when a cause of the problem is determined. Also, because 
the architecture allows for the multimedia information to be 
used in its original format, there is no need to re-engineer 
the media information for it to be deliverable. 

3 5 Based on this software architecture, there are two 

system utilities: the runtime solution or utility referred 
to as RUNTIME and the diagnostic tree building solution or 
utility referred to as BUILDER, as also described 
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hereinbelow. 
RUNTIME 

5 Within the RUNTIME software there are two forms of 

executing the diagnostics: PLC directed or operator 
step-by-step operation. The PLC directed approach involves 
the computer receiving a predefined PLC fault code relative 
to a press malfunction. Once the PLC fault is recognized, 

10 the software jumps to a specific assigned location in the 
diagnostic tree (the same diagnostic process can be 
completed using the operator step-by-step approach, 
discussed below) . The benefit of this approach is to 
minimize operator-to-computer diagnostic dialogue and get to 

15 the cause of the problem as quickly as possible. The key 
limitation of this approach is that it only applies to 
sensory faults. Non-sensory faults will need to be 
diagnosed in the step-by-step approach. 

In either of the approaches (PLC directed or step-by- 

20 step) , the operator is asked a series of questions until he 
reaches the cause of the press problem. In the step-by-step 
approach, the diagnostics always starts from the top of the 
tree. Answers for questions are presented in one of two 
formats: text or graphics with text. The text type answer 

25 is a traditional format illustrated in Figure 7, and is the 
easiest to generate and least time-consuming. The graphics 
with text, as illustrated in Figure 8, is a format that 
conveys the message faster and with less effort for the 
operator, but does take more development time because of the 

3 0 graphic development. Both of the answer formats can be 

intermixed in the diagnostics, but are defined and fixed in 
the diagnostic at the time of tree development. 

Another important feature in both the Question & Answer 
formats is the placeholders. The placeholder reminds the 

35 operator of the path that has been chosen and allows the 
operator to "Back Up" to a previous diagnostic screen. 

Also, at any time during the diagnostic session, if 
related machine specific support information has been 
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assigned to a particular question & answer or solution 
screen, an "i" button is displayed in the upper left hand 
corner as illustrated in Figures 7 and 8. By selecting the 
"i" button, the presentation server 26 is activated as 
5 illustrated in Figure 9, which allows the operator to view 
the different assigned media relative to that node function 
in the form of multimedia commands. More detail on the use 
of the runtime solution is described hereinbelow. 

10 BUILDER 

BUILDER allows one to define and/or maintain machine 
diagnostic trees (a graphical representation of a diagnostic 
decision step or flow) and assign multimedia machine 
15 information to the respective tree node. 

To briefly demonstrate the ease and simplicity of 
creating a diagnostic tree, reference may be made to Figure 
10. Diagnostic trees are made up of branches. Branches are 
made up of four primary components: 
20 * Question & Answer (Q&A) nodes 

* Solutions nodes 

* Branch jump nodes 

* links or lines. 

The Q&A node states a question with an unlimited number 
25 of answers. The solution node states the cause of the 

problem and needed repair for the cause of the problem. To ■ 
assign a Q&A node (or solution node) , a point mark is first 
located on the screen (click mouse) where one would like to 
place the node. Second, the type of node is selected from 
3 0 the above screen menu select: OBJECT-CREATE-SYMPTOM for Q&A 
node, or OBJECT-CREATE-SOLUTION for a solution node. This 
procedure places the object node on the screen as previously 
marked. To fill in the node name, the Q&A, or solution text 
information, the node is selected with the mouse pointer and 
35 the text is entered. Graphical answers with text can 
alternatively be used to replace the standard textual 
answers. To add more answers to a Q&A node, the node is 
selected and then OBJECT-ADD VALUE/TEST is selected from the 
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menu screen. To link an answer to the next question or 
solution , a right arrow icon is selected in the preferred 
answer box then the mouse pointer is dragged to the 
respective question, branch or solution and the mouse button 
5 is released. This places a link or line between the two 
object nodes. 

The above steps are repeated until the desired tree is 
built. Objects within a screen can freely be moved around, 
primarily for appearance and readability, without effecting 
10 the linking. Anytime while editing a tree, one can run the 
diagnostics by selecting from the screen menu, FILE-RUN 
TEST. This will bring one into the RUNTIME (test mode) with 
the current tree information.. 

To define machine specific multimedia information for 
15 any of the Q&A and solution nodes, one needs to access the 
presentation server screen. First, one double click on "?" 
in the Q&A box or just on the solution box. This brings up 
the question or solution editor screen and then one enters 
CTRL and -> keys, which brings up the presentation server 
20 (screen illustrated in Figure 11) . The keys do this by 
executing a recorded Windows 3.0 macro. To assign the 
desired media (documentation, drawing, photograph or video) , 
the button on the left is selected. A file directory box 
(not shown) pops up to allow one to look up the hard disk 
25 for the desired file. Once the file is selected, there is 
an opportunity to preview the material, define required 
additional information (i.e. page number within a 
documentation) and assign comment information before 
mapping. When the media is selected, it is listed in the 
30 appropriate media box as illustrated in Figure 11. There 
are, preferably, five different choices of material for each 
of the forms of media. When all the media is assigned, OK 
is entered to return to the builder screen. 
Building Diagnostic Trees And 
35 Assigning Multimedia Information 



- 11 - 



Using Graphic Icons 

The process here is similar to the above BUILDER 
process but is simpler to generate trees and assign 
5 multimedia information. There is also the use of more 
graphic icons (i.e. see the solution block of Figure 12) 
instead of menu selections. 



STEP 1: 



10 



To an assign symptom block (Q&A block) or solution 
block, an icon on the left in Figure 12 is first selected, 
and located on the right side or work area of Figure 12. To 
fill in the Q&A or solution block text information name, the 
15 block field with the mouse pointer is selected and the text 
is entered. To link an answer to the next question or 
solution, the link icon is selected and the respective 
blocks are connected. This places a link or line between 
the two blocks. 

20 The above steps are repeated until the desired tree is 

built. Blocks within a screen can freely be moved around, 
primarily for appearance and readability, without effecting 
the linking. 

The use of icons is a unique method for building 
25 diagnostic trees and defining multimedia information- Prior 
solutions required menu selections or involved accessing 
other screens. 



STEP 2 : 

30 



To define machine specific multimedia information for 
any of the Q&A and solution blocks, a media icon of choice 
is selected on the left side of the screen of Figure 12. 
This media icon is placed near a Q&A or solution block of 
35 choice and is then linked to the desired block. By 

reselecting an assigned media icon on the right side, a 
media selection screen pops up as illustrated in Figure 13 
Once the file is selected, there is the opportunity to 



preview the material, define required additional information 
(i.e. page number within a documentation) and assign comment 
information before mapping. 

This step is unique, because of the use of direct 
> visual aids (icons) and programming to assign the media to 
its respective block. This process is simpler than prior 
solutions because everything is accomplished on one screen, 
media remains in its natural format and does not require any 
re-engineering. 

.0 

STARTUP 

First one turns on the computer power or reboots the 
system. After a few minutes (software applications are 

5 being loaded) , the first screen of the system as illustrated 
in Figure 14 is displayed. This initial Question & Answer 
node is displayed on a tree display portion of the screen. 
This initial node requests one to select what transfer press 
subsystem (Feeder, Press or Destacker) has a problem. After 

0 selection in response to an initial command, additional 
Question & Answer screens are displayed until one reaches a 
solution screen or node. Details of each of the different 
types of screens and their operations are described 
hereinbelow . 

5 

QUESTION AND ANSWER SCREEN 

The diagnostic screen of Figure 15 is an example of a 
Question & Answer (Q&A) screen. The standard Q&A screen is 
D broken down into three major components: the question, the 
answer, and the placeholder. Also, in some of the Q&A 
screens, the Info Button is displayed, as highlighted in 
Figure 15, which means that there is additional multimedia 
information defined with the question. 



- 13 - 



QUESTION BOX 

For each diagnostic screen, there is a question posted 
and thus must be answered. The question is typically stated 
5 in text format only. 

ANSWER AREA 

In the answer area, there can be two types of answers: 
10 text or graphics with text as shown in Figure 15. The next 
type is just a sentence as shown in Figure 7. The graphics 
with text type shows a picture that represents the answer. 
To answer a question, the correct answer (line of text or 
picture) is selected. 

15 

PLACEHOLDER 

Once an answer is selected, a box with a picture or 
text, is placed in the placeholder area or branch display 
20 portion of the tree display portion of the screen. These 
placeholder boxes represent the diagnostic path that has 
been taken. Selecting a placeholder box allows one to 
"BACKUP" to a previous diagnostic screen. 

25 INFORMATION 

By selecting the Info Button, "i", the multimedia 
presentation screen pops up, as shown in Figure 9. This 
screen allows one to select desired media information 
30 (documentation, photographs, prints/drawings and full motion 
video) in response to the received user command. The 
multimedia information is there for the operator to help 
answer any questions or to explain testing or repair 
procedures about the current diagnostic screen. 
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View Screen For Text Document 

The screen of Figure 17 displays a text document, which 
can have embedded graphics, for a respective diagnostic 

5 screen. The whole page is not displayed. To view the other 
portions of the page, the arrow buttons (highlighted) are 
selected to move about the page. To look at the previous or 
next page within the document, the page symbol at the right 
hand side of the screen is selected (as highlighted) . 

10 Finally, to exit and return back to the diagnostic screen 
"File" is selected from the menu and then "Done" is chosen 
from the "File Menu." 

View Screen For Drawing 

15 

The screen of Figure 18 displays an AutoCad drawing 
file (also called a print) and predefined views within an 
AutoCad file. These drawings can be an electrical, 
pneumatic, hydraulic or mechanical drawing. To exit the 
2 0 drawing, the screen is touched. 

View Screen For Still Video 

By selecting any of the still video (same as a 
25 photograph) buttons (the camera icon) , the image is 

displayed in the top left side of the screen. An associated 
comment about the still video may be displayed above the 
still buttons as illustrated in Figures 16 and 19. 

30 View Screen For Full Motion Video 

By selecting any of the full motion video buttons (the 
video tape icon) , the video image is displayed in the top 
left side of the screen as illustrated in Figure 20. Prior 
35 to displaying the video, a comment is displayed at the top 
of the box. This gives the operator the chance to determine 
if it is the right video to view. If not, another video 
button is selected. If it is the right video, the VIEW 
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button is selected and the VCR will start searching for the 
tape segment under control of the computer- Once the video 
is found, it is displayed in the upper left side of the 
screen. Volume (if speakers are attached) and brightness 

5 can be adjusted while monitoring video tape. Two other 
video control buttons allow the audio to be muted and the 
video to be frozen (the VCR is still playing at this time 
and only the image on the screen is frozen) - To exit once 
the video is completed, the DONE button is selected in the 

10 video control box. 

Solution Screen 

When the last node, or the Solution, has been reached, 
15 the screen of Figure 21 is displayed. If there is any help 
information assigned at this node, the help information is 
displayed first. At this point, one can still back up if 
needed or save this particular diagnostic session in a 
session file (*.SES). 

20 

To save session 

To save current session from the top to current node, 
File-Save Session menu is selected. The appropriate file 
25 name is typed in. The default file extension is (*.SES). 

To restore session 

To restore a previously saved diagnostic session, File- 
30 Restore Session menu is selected. Then, the session file 
name is selected. By restoring session, one can trace the 
previous path or immediately return to the solution node. 

The system as described herein below diagnoses three 
primary components of a machine tool such as a transfer 
35 press: the press, the feeder, and the destacker. The press 
and the feeder diagnostic trees and related information are 
included. The press and feeder trees may address as many as 
143 primary symptoms that include approximately 800 nodes. 
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Building these two trees and accumulating and assigning the 
multimedia information is relatively straight forward after 
the paper, drawings and documentation were converted into 
electronic format . 
5 The method and system can be used to visually locate 

components on a large and complex transfer press. This has 
a significant impact in reducing the repair time because 
plant electricians can immediately locate defective 
component (s) . 

10 The method and system is not limited to stamping press 

diagnostics, but may be used for other forms of diagnostics 
(i.e. other plant machinery and office applications) and for 
use on non-machine diagnostics (i.e. process diagnostics or 
as engineering design aid) . 

15 The method and system provide the following advantages 

for use with a press: machine diagnostics when the machine 
is delivered; better transfer of technology and 
documentation from the vendor to the customer; reduced press 
down time; reduced diagnostic and repair time; better 

20 informed repair personnel, especially the apprentice; on- 
line training; better utilization of vendor/machine 
information; better service to the machine; the opportunity 
to capture new and refined diagnostic knowledge from plant 
personnel; reusability of information, that can be applied 

25 to other similar machines; and actual machine diagnostic 
flow and cause information can be captured to provide good 
feedback to the supplier with the hope of making better 
equipment and diagnostic models in the future. 

The method and system can be used to provide a multi- 

30 lingual solution. Using a stereo VCR, one audio track may 
contain information in a first language and the other audio 
track may contain information in a second language. 
Further, a multi-lingual solution is made possible through 
the selection of alternate keyboard assignments using the 

35 windows operating environment. 

The present system requires well disciplined and 
structured information management techniques to allow for 
ease of upgradeability , maintenance and adjustments. This 
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process is also invisible and seamless to the people making 
these changes. 

The presented media information demands high storage 
space but can be used for other similar and identical 

5 machines. To optimize storage and decrease redundancy of 
information, there may be a centralized network/ file server 
solution. While there is currently no clean way to store, 
retrieve and transmit analog information (video and audio) 
over a data network, a complete centralized library of 

10 machine information such as 2 and 3 dimensional animation, 
mult i- language media, audible answers to diagnostic 
questions, speech, 3-D drawings, and a point-to-point 
distribution system can be accommodated when such is 
possible . 
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CLAIM 

1. A method for processing and presenting multimedia 
information in a tree structure including branches having a 
5 plurality of nodes and node-connecting links in a computer 
system having tree display means for displaying nodes of the 
tree structure, the method comprising the steps of: 

providing command display means for displaying 
multimedia commands representative of various types of 
10 multimedia information; 

providing multimedia output means for providing 
multimedia information to a user of the systems- 
providing a knowledge base for storing a 
multiplicity of information records mapped into the tree 
15 structure, the information records including: 

means for denoting the text strings that can be 
displayed on the tree display means; 

means for denoting the multimedia commands that 
can be displayed on the command display means; and 
20 means for denoting the multimedia information that 

can be presented by said multimedia output means; 
receiving an initial command; 

displaying at least one selected node of the tree 
structure including at least one text string on the tree 
25 display means in response to the initial command; 

displaying a plurality of multimedia commands 
corresponding to the at least one selected node on the 
command display means; 

receiving a user command related to one of the 
30 displayed multimedia commands; and 

delivering a selected amount of the multimedia 
information to the multimedia output means to be presented 
thereby to the user of the system in response to the user 
command. 

35 
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2. A system for processing and presenting multimedia 
information in a tree structure including branches having a 
plurality of nodes and node-connecting links in a computer 
system having tree display means for displaying nodes of the 
5 tree structure, the system comprising: 

command display means for displaying multimedia 
commands representative of various types of multimedia 
information; 

multimedia output means for providing multimedia 
10 information to a user of the system; 

a knowledge base for storing a multiplicity of 
information records mapped into the tree structure, the 
information records including: 

means for denoting the text strings that can be 
15 displayed on the tree display means; 

means for denoting the multimedia commands that 
can be displayed on the command display means; and 

means for denoting the multimedia information that 
can be presented by said multimedia output means; 
20 means for receiving an initial command, the tree 

display means displaying at least one selected node of the 
tree structure including at least one text string in 
response to the initial command, and the command display 
means displaying a plurality of multimedia commands 
25 corresponding to the at least one selected node on the 
command display means; and 

means for receiving a user command related to one 
of the displayed multimedia commands, the multimedia output 
means presenting a selected amount of the multimedia 
30 information to the user of the system in response to the 
user command. 



3. The invention as claimed in claim 1 or claim 2 
wherein the knowledge base references at least a portion of 
35 the multimedia information stored in analog form. 
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4. The invention as claimed in claim 1 or claim 2 
wherein the knowledge base stores at least a portion of the 
multimedia information in digital form. 

5 

5. The invention as claimed in claim 1 or claim 2 
wherein the computer system is an expert computer system. 

6. The invention as claimed in claim 1 or claim 2 

10 wherein at least a portion of the multimedia information is 
diagnostic information . 

7. The invention as claimed in claim 1 or claim 2 
wherein the at least one selected node is a question and 

15 answer node. 

8. The invention as claimed in claim 1 or claim 2 
wherein the nodes of the tree structure include question and 
answer and branch jump nodes. 

20 

9. The invention as claimed in claim 1 or claim 2 
wherein the nodes of the tree structure include question and 
answer, branch jump and solution nodes. 

25 10. The invention as claimed in claim 1 or claim 2 

wherein the tree structure is a diagnostic tree structure. 

11. The invention as claimed in claim 1 or claim 2 
wherein the multimedia output means includes video display 

30 means for displaying video images. 

12. The invention as claimed in claim 1 or claim 2 
wherein the multimedia output means includes sound 
generating means for generating sound messages. 

35 

13. The invention as claimed in claim 1 or claim 2 
wherein the multimedia output means includes drawing display 
means for displaying drawing images. 
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14. The invention as claimed in claim 1 or claim 2 
wherein the multimedia output means includes graphics 
display means for displaying graphical images. 

5 15. The invention as claimed in claim 1 or claim 2 

wherein the multimedia output means includes document 
display means for displaying images of documents. 

16. The invention as claimed in claim 1 or claim 2 
10 wherein the knowledge base stores a multiplicity of 

digitized video images, each of the stored video images 
corresponding to one or more pieces of equipment from a 
predefined set of pieces of equipment. 

15 17. The invention as claimed in claim 1 or claim 2 

wherein the multimedia commands are displayed on the command 
display means in the form of graphic icons. 

18. The invention as claimed in claim 1 or claim 2 
2 0 wherein the initial command is a user command. 

19. The invention as claimed in claim 1 or claim 2 
wherein the tree display means, the command display means 
and the multimedia output means define a video display of 

25 the computer system. 

20. The invention as claimed in claim 1 or claim 2 
wherein the tree display means includes branch display means 
for displaying the selected nodes of the tree structure. 



30 



35 



21. The invention as claimed in claim 1 or claim 2 
wherein the information records further include means for 
denoting graphical images that can be displayed on the tree 
display means. 

22, A method for creating a tree structure for 
processing and presenting multimedia information, said tree 
structure including branches having a plurality of nodes and 
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node-connecting links in a computer system having tree 
display means for displaying nodes of the tree structure, 
the method comprising the steps of: 

providing a computer system, the computer system 

5 including : 

command display means for displaying multimedia 
commands representative of various types of the multimedia 
information; 

multimedia output means for providing the 
10 multimedia information to the system; 

multimedia input means for providing the 
multimedia information to the system; 

multimedia storage means for storing the 
multimedia information; 
15 inputing the multimedia information to the 

computer system; 

storing the multimedia information; 

providing a knowledge base for storing a 
multiplicity of information records mapped into the tree 
20 structure, the information records including: 

means for denoting text strings that can be 
displayed on the tree display means; 

means for denoting multimedia commands that can be 
displayed on the command display means; and 
2 5 means for denoting the multimedia information that 

can be presented by said multimedia output means; and 

assigning the stored multimedia information to one 
or more of the information records. 

30 23. A system for creating a tree structure for 

processing and presenting multimedia information, said tree 
structure including branches having a plurality of nodes and 
node-connecting links in a computer system having tree 
display means for displaying nodes of the tree structure, 
35 the system comprising: 

a computer system, the computer system including: 
command display means for displaying multimedia 
commands representative of various types of the multimedia 
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information; 

multimedia output means for providing the 
multimedia information to a user of the system; 

multimedia input means for providing the 
5 multimedia information to the system; 

multimedia storage means for storing the 
multimedia information in the form of a knowledge base, the 
knowledge base storing a multiplicity of information records 
mapped into the tree structure, the information records 
10 including: 

means for denoting text strings that can be 
displayed on the tree display means; 

means for denoting multimedia commands that can be 
displayed on the command display means; and 
15 means for denoting the multimedia information that 

can be presented by said multimedia output means; and 

means for assigning said stored multimedia 
information to at least one information record. 

20 
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